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Abstract 


In  the  1997  PTTI  paper  titled  “ Maintenance  ofHP5071A  Primary  Frequency 
Standards  at  USNO,  ”  Chadsey  and  Kubik  presented  findings  to  that  date  on  USNO’s 
efforts  to  evaluate  the  devices  ’  performance  according  to  their  internal  operational 
parameters.  This  paper  presents  methods  used  to  evaluate  those  parameters  and 
automatically  detect  abnormal  operation.  Some  of  the  developmental  difficulties  will  be 
discussed,  as  will  some  general  guidelines  on  the  tolerances  USNO  uses  to  detect 
problems  with  the  HP  5071 A  frequency  standards. 

INTRODUCTION 

The  importance  of  monitoring  of  the  operating  parameters  of  the  HP  5071 A  cesium 
frequency  standard  was  covered  in  a  1997  PTTI  paper  by  Chadsey  and  Kubik  [1].  It  was 
noted  that  by  having  a  computer  periodically  inquire  and  permanently  file  the  22 
parameters  via  the  RS-232  connection,  one  could  diagnose  and  in  some  cases  predict  the 
device’s  failure.  All  the  data  analysis  was  performed  manually  when  that  first  paper  was 
written.  Manual  analysis  may  be  acceptable  for  some  locations  which  have  only  a  few 
HP  5071 A  devices,  but  when  five  or  more  devices  are  involved,  the  personnel  costs  go  up 
very  quickly.  The  need  to  regularly  look  at  the  parameters  for  over  40  devices  at  USNO 
in  Washington,  DC  drove  the  development  of  automating  the  data  analysis  toward  an 
alarm  system.  Warnings  were  to  be  issued  whenever  a  device  exhibited  abnormal 
behavior. 

THE  PARAMETERS 

A  computer  can  be  programmed  to  periodically  poll  an  HP  5071 A  via  the  RS-232 
connection.  The  parameters  reported  are:  frequency  offset,  oscillator  control  percentage, 
rf  1  amplitude  percentage,  rf  2  amplitude  percentage,  Zeeman  frequency,  C  field  current, 
electron  multiplier  voltage,  signal  gain  percentage,  tube  oven  voltage,  tube  oven 
temperature  error,  oscillator  oven  voltage,  ion  pump  current,  hot  wire  ionizer  voltage, 
mass  spectrometer  voltage,  SAW  tuning  voltage,  DRO  tuning  voltage,  87  MHz  PLL 
voltage,  uP  clock  PLL  voltage,  +12  volt  supply  voltage,  -12  volt  supply  voltage,  5  volt 
supply  voltage,  and  internal  temperature.  A  detailed  description  of  these  is  found  in  the 
operators  manual  and  the  1997  paper  [1],  These  parameters  tell  a  person  how  the 
hardware  of  a  device  is  operating.  A  hardware  failure  may  or  may  not  be  noticeable  in  a 
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device’s  performance.  If  the  failure  affects  performance,  the  question  becomes:  how 
long  before  it  is  noticeable?  A  power  supply  failure  usually  will  show  up  immediately  in 
the  performance,  while  a  cesium  beam  tube  end  of  life  may  not  show  up  for  several 
weeks,  depending  on  the  accuracy  of  the  measurement  system. 

THE  BASIC  PROGRAM 

The  original  program  used  at  USNO  to  monitor  the  HP  5071 A  parameters  was  quite 
basic.  Its  first  duty  was  to  take  the  parameters  as  they  were  filed  and  put  them  in  a  more 
readable  form.  Several  different  formats  were  required  depending  on  whether  a  person 
was  to  look  at  the  data  or  the  newly  formatted  data  were  to  be  used  as  an  input  into  a 
plotter  or  spreadsheet  program. 

From  this  first  reformatting  program,  the  next  logical  step  is  to  have  the  program  check 
each  parameter  for  reasonableness.  If  a  measurement  exceeds  a  certain  limit,  a  warning 
message  is  printed.  If,  as  in  this  case,  there  are  several  parameters  which  have  to  be 
checked,  one  has  to  write  a  series  of  such  tests.  Such  a  test  usually  has  both  upper  and 
lower  limits.  For  example,  a  power  supply  can  fail  and  stop  working  if  a  capacitor  blows 
open.  On  the  other  hand,  if  the  capacitor  blows  to  a  short,  the  voltage  could  go  very  high, 
causing  damage  in  other  components.  Thus,  the  programmer  needs  to  write  the  source 
code  with  the  ability  to  check  22  different  ranges  of  values  for  the  HP507 1 . 

However,  the  parameter  range  for  any  two  devices  may  be  and  usually  is  different.  The 
difficulty  is  that  the  difference  between  the  ranges  of  the  two  devices  is  larger  than  the 
tolerance  for  catching  the  device  failures.  For  example,  one  device  may  be  performing 
very  well  with  the  typical  ion  pump  current  of  0.5  or  less  microamps,  as  shown  in  Figure 
1. 


HP0166  Ion  Pump  Current 
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Figure  1 .  Typical  Ion  Pump  current  for  an  HP507 1  Cesium-Beam  Frequency  Standard 
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Another  device,  however,  can  be  performing  equally  well  while  running  an  ion  pump 
current  over  1 .0  microamp,  as  shown  in  Figure  2. 


51420  51430  51440  51450 

MJD 


Figure  2.  A  typical  Ion  Pump  current  for  an  HP507 1  Cesium-Beam  Frequency  Standard. 
Although  the  current  is  above  the  typical  value  and  is  not  well  regulated,  the  device  is  performing 
as  well  as  the  one  shown  above.  This  device  must  be  closely  watched  for  performance  changes. 


The  two  devices  show  no  difference  in  their  frequency  stability,  as  demonstrated  by  their 
sigma-tau  plots  shown  in  Figure  3.  To  allow  for  this  diversity,  the  programmer  must 
either  write  a  separate  program  to  check  the  parameters  of  each  device  or  create  a 
configuration  file  to  let  the  program  know  the  tolerances  for  each  device.  The 
configuration  file  idea  is  really  not  difficult  if  one  uses  matrixes  to  store  the  limits  for 
each  device’s  parameters. 
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CHANGES  IN  PARAMETERS 


Up  until  now,  all  parameters  have  been  discussed  as  though  they  only  range  within  fixed 
limits  and  have  no  trends  that  would  change  the  limits.  For  example,  a  +12  volt  power 
supply  would  always  be  between  +12.0  and  +12.5  volts.  Any  deviation  from  that  range 
would  indicate  a  performance  affecting  failure  of  some  type.  While  this  is  true  for  power 
supplies  and  most  of  the  other  parameters,  it  is  not  true  for  all  of  them.  The  best  example 
is  the  electron  multiplier  voltage. 

The  electron  multiplier  (EM)  voltage  changes  as  the  cesium  clock  ages.  When  the  tube  is 
new,  the  EM  voltage  will  start  out  at  some  value,  say  1600  volts.  Over  the  first  6  months 
to  a  year  that  voltage  will  slowly  decrease  as  much  as  200  to  300  volts.  After  bottoming 
out,  the  EM  voltage  will  slowly  increase  as  the  tube  ages.  Because  the  value  changes 
slowly  over  time,  fixed  limits  that  are  too  narrow  will  eventually  be  exceeded  and  an 
error  reported  when  in  fact  one  did  not  occur.  If  the  limits  are  set  too  wide,  a  real  error 
will  go  undetected.  The  program  must  be  able  to  learn  what  is  normal  before  determining 
when  the  parameter  has  gone  beyond  normal  operating  tolerances  (see  Figure  4). 
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Figure  4.  A  typical  Cesium-Beam  Frequency  Standard  has  a  slowly  varying  Electron  Multiplier  voltage 
which  the  alarm  program  must  track  to  detect  a  sudden  change.  The  >9.0  volt  change  occurring  beginning 
on  MJD  51449  is  the  indicator  that  the  cesium  supply  has  been  depleted  in  this  tube.  Performance  will  start 

to  degrade  sometime  in  the  next  few  days  to  weeks. 


The  challenge  of  writing  a  self-learning  program  involves  understanding  and 
implementing  one’s  mental  analytical  techniques.  This  is  almost  always  easier  said  than 
done.  One  must  consider  how  long  a  period  of  time  must  be  used  to  determine  normalcy. 
If  the  period  is  too  short,  the  noisiness  of  the  data  may  hide  the  abnormal  events  the  alarm 
program  is  supposed  to  be  detecting.  If  the  period  of  time  is  too  long,  long-term  data 
trends  may  induce  the  appearance  of  noise  or  may  cause  the  program  to  ignore  a  subtle 
yet  significant  change  affecting  performance.  While  determining  the  proper  period  for 
data  analysis,  one  must  estimate  the  expected  range  of  variation  for  normal  data. 
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Many  hours  of  manual  data  analysis  and  experimentation  were  necessary  to  find  criteria 
that  could  be  applied  to  all  the  parameters.  (A  programmer  usually  wants  to  make  the 
source  code  as  short  and  universal  as  possible.)  The  present  alarm  program  uses  the  data 
collected  over  a  5-day  period  and  linear-fitted  to  determine  normality.  It  was  found  that 
data  analysis  periods  of  more  than  7  days  required  the  tolerance  limits  on  some 
parameters  to  be  too  large  to  catch  abnormal  behavior  in  the  devices.  Shorter  analysis 
periods  tended  not  to  average  out  the  measurement  noise  sufficiently  for  some 
parameters.  The  program  compares  a  theoretically  normal  point  to  the  actual  measured 
value.  An  alarm  occurs  if  they  differ  by  some  a  priori  amount  which  must  be 
empirically  determined.  How  far  into  the  future  the  linear-fit  solution  point  must  be  is 
controlled  by  the  rate  at  which  a  parameter  can  change  during  normal  operation.  Again, 
if  one  does  not  look  the  proper  distance  into  the  future,  a  false  alarm  can  occur.  The 
present  program  predicts  48  hours  in  advance.  While  24  hours  was  sufficient  for  most 
parameters,  it  would  not  work  for  three  very  important  parameters  (i.e.,  RF  amp  #1,  RF 
amp  #2,  and  electron  multiplier  voltage).  The  prediction  time  was  made  the  same  for  all 
parameters  to  make  the  program  as  simple  yet  universal  as  possible. 

PARAMETER  TESTING 

Next,  we  consider  what  the  program  should  deem  abnormal  and  cause  an  alarm  message 
to  be  generated.  At  least  two  levels  of  alarm  conditions  were  found  to  be  necessary.  The 
lower  level  is  a  warning  that  something  might  be  failing,  or  had  just  failed,  but  had  not 
yet  affected  the  device  performance.  The  higher  level  alarm  warns  of  performance 
affecting  failure.  For  example,  a  +12  volt  power  supply  changing  0. 1  or  0.2  volts  may 
not  affect  performance  or  clock  operations,  but  could  indicate  that  something  could  be 
starting  to  fail.  On  the  other  hand,  a  drop  in  the  power  supply  voltage  from  +12  volts  to 
+0.2  volt,  as  shown  in  Figure  5,  would  affect  the  device’s  performance  and  operation, 
requiring  the  device’s  immediate  repair. 


HP  0394  +12  Volt  P  Okve  r  Supply 
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Figure  5.  A  performance  affecting  drop  in  the  +12  volt  power  supply  in  an  HP  5071  Cesium-Beam 

Frequency  Standard. 
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As  stated  earlier,  each  device  has  a  “personality”  of  its  own.  No  two  devices  have 
exactly  the  same  operating  characteristics  for  all  the  parameters.  However,  some  of  the 
parameters  (i.e.,  Zeeman  frequency  and  signal  gain)  have  the  same  expected  values  and 
tolerances  in  all  the  devices.  To  make  the  program  as  universal  as  possible,  it  was 
decided  to  treat  all  parameters  the  same  and  have  low- and  high-level  alarm  limits  for  all 
of  them.  This  gives  the  added  benefit  of  making  each  device’s  characteristics  file 
somewhat  self-documenting.  A  parameter  (e.g.,  ion  pump  current)  may  be  different  for  a 
particular  device  and  the  device  still  be  operating  and  performing  perfectly  well  for  a 
site’s  timing  operations.  The  following  table  gives  the  characteristics  of  a  typical 
HP5071. 


HP0101-5071 

HP0101 

78_125 

cesiurn/ClOl 

51513.399 

0.0000 

0 . 000 

0 . 000 

0.000 

f/s 

Freer  offset 

F_of  f 

0.2779 

0 . 100 

-20.000 

45.000 

pc 

Osc_control 

0_ctl 

31.0212 

0.100 

15.000 

35.000 

pc 

RF_amp_l 

RF_1 

29.6668 

0.100 

15.000 

35.000 

pc 

RF_amp_2 

RF_2 

39949.0000 

0 . 000 

39949 . 000 

39949 . 000 

Hz 

Zeeman_f req 

Z_f  rq 

12.1133 

0.001 

12 . 100 

12.200 

mA 

C-f ield 

C_f  Id 

1357 . 5823 

9 . 000 

0 . 000 

2552 . 000 

V 

E_multiplier 

E_mlt 

14.4000 

0.000 

14.400 

14.400 

pc 

Signal_gain 

S_gn 

7.5279 

0.100 

7 . 000 

9.000 

V 

CBT_oven 

C_ovn 

0.0032 

0.200 

-0.200 

0.200 

c 

CBT_error 

C_err 

-8.7000 

0 . 100 

-9.000 

-8.500 

V 

Osc_oven 

O_ovn 

0.2040 

0.200 

0.000 

0.500 

UA 

Ion_pump 

I_pmp 

1.0042 

0 . 100 

0.900 

1.100 

V 

HW_ionizer 

HW 

11.7000 

0 . 100 

10.000 

14.000 

V 

Mass_spec 

Mass 

-0.9000 

0.100 

-2.500 

2.500 

V 

SAW_tuning 

SAW 

5.5000 

0.050 

5 . 000 

7.000 

V 

DRO_tuning 

DRO 

1.2964 

0.100 

1.000 

3.500 

V 

87MHz_PLL 

87PLL 

3.2000 

0 . 050 

2.500 

3.500 

V 

Up_clock_PLL 

UpPLL 

12.3000 

0.100 

12.000 

12.500 

V 

+12V_supply 

P12V 

-12.1042 

0 . 100 

-13 . 000 

-12.000 

V 

-12V_supply 

M12V 

5.2052 

0.100 

5.000 

6.000 

V 

+5V_supply 

P5V 

43 . 1294 

0.400 

30 . 000 

50 . 000 

c 

Thermometer 

Temp 

Tba  fd  lirta  a  nnmA  / «  a 

iiiv  mol  iiiiw  isVJiiLU.1110  uiw  uvviwv  o  vv/liipiv>tfc<  name 


the  device),  an  abbreviation  of  the  device  name,  and  the  device’s  location  (i.e.,  USNO  Washington  building 
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projected  points  were  last  calculated.  The  rest  of  the  file  contains  the  specifics  about  each  parameter. 
(Note:  USNO  does  not  put  frequency  adjustments  into  its  clocks.  Device  rates  and  drifts  are  accounted  for 
through  computations  and  not  by  hardware  adjustments.  Thus,  frequency  offset  is  set  to  zero  in  all  USNO 
clocks.) 


Each  parameter  characteristic  line  is  calculated  independently  and  then  assembled  into 
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expected  values  (the  2-day  prediction  from  the  5-day  linear  fit  to  the  data).  The  second 
field  is  the  low  level  alarm  level. 


level  alarm  tolerance  added  to  the  predicted  value,  a  warning  is  issued.  When  the  low- 
level  alarm  value  is  zero,  an  alarm  occurs  for  any  deviation  from  the  first  field  value. 

The  third  and  fourth  fields  are  the  minimum  and  maximum  tolerable  levels  for  the 
parameter  respectively.  If  the  parameter  value  should  go  outside  these  values,  the  device 
probably  is  experiencing  a  failure  that  could  affect  its  performance.  The  fifth  field  is  the 
units  of  measure.  The  sixth  field  is  the  parameter  name  and  the  seventh  field  is  the  name 
abbreviation.  The  parameter  name  and  name  abbreviation  are  used  in  different  alarm 
messages. 

RESULTS 

Over  the  past  two  years,  this  program  has  given  some  minor  false  alarms.  Those  alarms 
were  caused  by  the  device’s  incorrect  output  values;  these  are  usually  one-point 
anomalies.  The  linear  processing  done  to  learn  the  normal  operational  value  of  each 
parameter  does  not  reject  these  anomalies,  but  warning  messages  about  them  are  issued. 
The  operator  must  double-check  that  there  is  no  real  problem.  One  must  remember  that 
this  is  an  alarm  program.  Outlying  points  should  not  be  filtered  out,  as  they  would  in 
most  data  processing.  This  should  be  noted  by  users  of  any  automated  alarm  system. 

The  checks  are  only  as  valid  as  the  data  quality  is  good.  Data  verification  is  still 
required. 

CONCLUSIONS 

A  program  has  been  written  at  USNO  to  automatically  check  the  operational  parameters 
available  from  an  HP  507 1  model  cesium-beam  frequency  standard.  The  program  warns 
operators  if  a  parameter  does  not  meet  marginal  or  critical  tolerances  which  might  or  will 
affect  the  device’s  performance.  It  is  sophisticated  enough  to  learn  of  slowly  changing 
parameter  values  in  order  to  better  detect  and  report  abnormal  values. 
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